…uscatable', fix mainMenu.obfuscate references (#385)

* 5.0 initial changes (#274)

* run black and isort

* Socketio reimplemented for 5.0 (#285)

* stub tests for startup loaders, convert bypass loader to bypass service

* 5.0 Download API (#290)

* Initial 5.0 testing (#291)

* 5.0:  Logging (#307)

* loggers

* initial replacing pydispatch and converting print statements to logs

* moving some things around replacing more print statements

* more logging setup.

* config, command line, and tests

* tests

* more work on agent logs

* more doc updates

* more cleanup

* refactoring for logging configs to work properly

* convert more listeners

* more listener conversion

* finish converting listeners to use logger

* cleanup

* ignore_errors on rmtree

* fix issues from 4.5.0 merge

* update submodules to match sponsors-dev

* convert to new config format

* 5.0 - More cleanup (#328)

* remove duplicate add_agent_task_db method

* pass db to hooks

* convert reporting to a plugin

* remove the prompt toolkit from the server

* changelog

* Starkiller submodule 2 (#329)

* add starkiller-sponsors submodule

* change remote starkiller

* checkout 2.0.0-alpha2

* fix other submodules

* add log for starkiller link

* use release token for private repo submodule

* fix a warning to see if it gets the test passing

* make bypass name conflict test more dynamic

* assert

* add relese_token to docker image build

* 5.0 Obfuscation (#340)

* Authors rework (#354)

* start authors. rename PydanticModule

* use ruamel for the conversion

* convert yamls

* stager updates

* fix covenant module load

* fix test_modules capsys -> caplog

* update plugin endpoints

* add a few missing links

* changelog

* increase line length on the yamls

* use alpha3

* 5.0 Plugin api (#358)

* add plugin api tests

* plugin error handling

* cleanup

* fix staging issue

* fix tests after 4.6 merge. Still failing to shut down after running. Check for changes in plugins from 4.6

* fix the hanging test issue

* don't instantiate main unless we are actually starting up

* 5.0 - Fix filter multi param (#371)

* Fix issue with the internal filters which were not returning all their params back to be passed to the next filter

* update multi_param test

* fix enum serialization

* use ObfuscationConfig for csharp. use ge/le instead of gt/lt for jitter. (#377)

* use ObfuscationConfig for csharp. use ge/le instead of gt/lt for jitter.

* remove .python-version file

* fix test

* add lifespan param to uvicorn to show lifespan errors, fix middleware issue that was breaking lifespan hooks, add shutdown event handler (#379)

* change python dep caching (#380)

* change python dep caching

* Update .github/workflows/lint-and-test.yml

* Update .github/workflows/lint-and-test.yml

* empty

* Client updates for 5.0 (#370)

* updated login to jwt

* updated listener creation

* generate stager works

* fixed autocomplte for stagers

* plugin updates

* fixed issue when recursively cloning

* removed csharp_exe listener check

* updated stager data to bytes

* fixed module execution

* fixed shell tasking

* fixed plugins

* fixed user management

* fixed enable/disable user

* removed client report endpoint

* updated malleable endpoints

* updated history and view tasks

* file download/upload needs work

* fixed notifications for tasks

* removed legacy notes until new version is built

* updated file upload

* found issue with download endpoint

* added comments for todos

* fixed listener list

* updated editlistener menu

* updated listener edit and kill

* fixed formatting

* fixed view and remove credentials

* added decode for tasking when in bytes

* fixed agent upload with directory limitiation

* fixed kill agent

* proxy endpoints missing

* fixed agent rename

* fixed shortcuts

* fixed vnc

* fixed view task

* caps for output

* removed unused functions

* fixed active agents displayed

* fixed hide stale agents

* formatting

* fixed csharp compiler error for obfuscation

* fixed vnc port error

* Update empire/client/src/menus/UseListenerMenu.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/client/src/menus/UseListenerMenu.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/client/src/menus/UseMenu.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* fixed preobfuscation

* changed preobfuscate format

* reverted test accidental test removal

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* remove commented reset db code

* remove the reporting files on reset

* 5.0 - Deprecating functions, finish proxy task endpoint. (#384)

* Mark credential and agent functions deprecated. add search to credential api

* add search to credentials

* proxies

* reuse the tasks service for get_queued_agent_tasks

* bump to starkiller v2.0.0- alpha4

* fix tests

* add a list endpoint for global obf configs, mark languages as 'preobfuscatable', fix mainMenu.obfuscate references (#385)

* 5.0 API Fixes  (#387)

* add 400 response to openapi spec, standardize router config, extend jwt expiration, wrap module generate so it doesn't throw 500

* fix import sort

* alpha4

* 5.0 api cleanup (#388)

* Refactor the api endpoints to be more consistent

* add author to the bypass endpoints

* remove a couple todos

* Shell command updates (#391)

* add a 'literal' flag to shell commands to ignore the aliased cases

* update python agent to handle the --literal flag

* 5.0 - Plugin notes and other todos (#397)

* add notes about 5.0 plugins and resolve some more todos

* rename v2beta in uri to v2

* remove more todos

* fix tests to properly use test config. Programatically add unique constraint for credentials

* remove print statements from plugin

* starkiller alpha5

* starkiller alpha6

* merge fixes

* Make plugins and new bypass 5.0 compatible

* 4->5 plugin notes

* Make the option handling code easier to follow, default values when required option not provided, combine module and listener/stager/plugin option handling (#409)

* add task search filter (#410)

* Convert server-side print to log messages (#406)

* removed prints from plugins

* added logging to multi/launcher

* more stager upodates for logging

* Update empire/server/modules/python/privesc/osx/dyld_print_to_file.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* moved to log to module level

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update to generate stageless agents (#407)

* database lock issue

* database lock on response

* database lock on response

* removed self.lock on response

* agent checks in - need to add sysinfo to client commands

* update sys info does not work

* formatting

* fixd database lock issue

* error during stageless exe generation

* fixed embedded stager

* updated python stageless

* moved generate agent to stagers

* formatting

* reverted changes

* removed ironpython comments

* fix some of the failing tests

* fix the option_util after 5.0-dev merge

* format

* Update empire/server/common/stagers.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/listeners/http.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* revert hooks change

* formatting

* Update empire/server/stagers/windows/csharp_exe.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/stagers/windows/generate_agent.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* made hooks update for empty array

Co-authored-by: Vince Rose <vrose04@gmail.com>

* Fixed additional todos from Client (#411)

* fix credential endpoints

* fixed agent checkin notification

* fixed script import

* fixed script command

* formatting

* remove check for external agent module (#412)

* remove check for external agent module

* add missing processes router

* fix serializable user error

* update plugin execution response and tests

* 5.0 - Agent response cleanup (#413)

* Reduce the amount of db calls in agent communications

* small optimization

* fix credential writes and change the way we check for uniqueness

* remove invalid semicolons

* fixed error when stageless is set for C# (#414)

* 5.0 - Moving files (#415)

* moving files around

* move starkiller submodule

* rename more files

* fixed reporting plugin and added options for reports (#416)

* 5.0 - Plugin execution updates (#417)

* update plugins to use

* dont modify params in validate_options

* update autostart_plugins function and add a test to validate

* black/isort example.plugin

* fix defaul detail str

* bump plugins

* bump starkiller to the sponsor version

* Move database under core/db, move invoke-obf under data/, move hooks … (#418)

* Move database under core/db, move invoke-obf under data/, move hooks under core

* change relative import

* invoke-obf location in dockerfile

* Move plugin_socketio_message and remove mainMenu.directory (#419)

* removing directories from main_menu, moving plugin socket messages to plugin_service

* update plugins

* move startup to separate method

* fix typos

* changelog

* fix the rest of the plugin messaging.

* bump starkiller to first sponsor beta build

* Prepare README for general release and add flag for running api with https (#424)

* update the readme to prepare for a general release and add a flag for running the api with https

* use restport

* fix file saving issues introduced in previous update (#425)

* fix file saving issues introduced in previous update

* cast port to int

* custom generate wasn't returning result, ps filter was creating a sec… (#426)

* custom generate wasn't returning result, ps filter was creating a second db session

* remove unused import

* bump starkiller to v2.0.0-beta2-sponsors

* bump version

* updated socks and chisel plugins for 5.0 (#443)

* Added clear window command to client (#441)

* added clear window command to client

* updated os.system clear to prompt.toolkit

* Fix for malleable c2 listener (#437)

* added ignore for listener options for malleable c2

* updated import for typing

* removed any for listener options

* moved serialized profile from listener options

* Removed unused generate_agent module and fixed install script (#440)

* removed unused generate_agent module and fixed install script

* changed to python-socketio from websocket-client

* Added mouse support to client (#442)

* added mouse support to client

* move mouse support option to yaml

* fixed empty dict as default

* change bool to false

* formatting

* Added RunOF support (#447)

* split runof to 64 and 32 bit modules

* added beacon_func embedded resources

* updated submodule

* fixed

* renamed folders

* added pass for architecture mismatch

* fixed formatting

* updated name to inject_bof, combined modules, and updated shortcuts

* set mouse-support to default off since it turns off highlighting for copy/paste

* formatting

* added bof module test

* added sleep timer for csharpserver to generate

* added check for empirecompiler.dll and wait for generation

* Formatting

* switched test since github cant handle the compiler

* move bof file to a fixture

* Update empire/test/test_agent_task_api.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* removed unused functions

Co-authored-by: Vince Rose <vrose04@gmail.com>

* Full MySQL support (#431)

* make a few tweaks to get python agents to work on mysql

* get more tests passing against mysql

* update github action

* temporarily remove a test

* fix password for github mysql

* fix tests for mysql

* update other listeners and extend test time for ci

* fix download_api test. Add mysql to image_test

* change default back to sqlite for now

* Add MySQL to install script/tests. Optimize Dockerfile.

* add token to test_install_script

* check for running in docker

* || true

* rework the database config so it can be in a single file and overwritten by an env var.

* Fix language checks. Fix column types on tasking.

* Fix install script containers

* install script tweaks for kali

* use mariadb for kali

* MITRE ATT&CK Updates (#448)

* added mitre attack tactics and information to the database

* added mitre attack framework to listeners

* added tactics to client menu

* fixed error with filename

* fixed issue when listener starts up

* added tactic and subtechnique examples

* added subtechnique to module techniques

* formatting

* fix test_agent_task_api module

* undo try/catch for module loading

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Updated running list of changes from 5.0 (#450)

* updated running list of changes from 5.0

* Update CHANGELOG.md

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update CHANGELOG.md

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update CHANGELOG.md

Co-authored-by: Vincent Rose <vrose04@gmail.com>

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Bypass language, stale processes, keyword length (#452)

* add minimum requirements for keyword dto

* add language to bypass endpoints. Update ps hook to mark processes stale. add requirements to keyword dto

* add stale process to endpoint, fix int comparison

* delete hostprocesses after hook test

* bump starkiller to beta3

* Added Client logging (#449)

* added basic debug logs to client

* initial error logs displayed and info without color

* updated formatting for client log file

* added new log level - message

* updated client logging

* modified some server returns to print message instead of log

* Update empire/client/client.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Fixes for client logging (#453)

* added basic debug logs to client

* initial error logs displayed and info without color

* updated formatting for client log file

* added new log level - message

* updated client logging

* modified some server returns to print message instead of log

* Update empire/client/client.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* removed log.message

* updated to use config file for logging level

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Removing more log.message from client (#456)

* removing more log.message from client

* fixed starkiller version

* Use bold ansi format to make the log messages more readable (#455)

* Check git submodules on server startup (#454)

* Fixes for obfuscation in 5.0 (#465)

* fixed seek error on tempfiles

* fixed obfuscation in 5.0

* fixed miscopied yamls

* formatting

* reverted accidental deletions

* Added plugin error handling and logging during intialization (#476)

* added better logging for plugin initialization

* formatting

* 5.0 - Starkiller config (#477)

* add starkiller config properties and a sync command

* move the starkiller sync to its own script

* refactor

* revert db password

* update test server config

* change killed to archived

* fix test_agents.py test

* fix tests again

* remove db files that were accidentally added

* skip stale expression test when not using sqlite

* propogate database_use env var to config

* use verbose pytest output

* add timeout to reset tests

* move submodule check

* close all db conns

* pass the config dict to the sync function (#480)

* In-band SOCKS Proxy (#423)

* created seperate background task for vnc

* secretsocks out of band

* fixed out of band socks

* task not written to database

* taskings sent but not entering queue on agent socks

* fixed in band comms - still needs clean up

* added pysecretsocks to poetry and renamed socks functions

* fixed task_socks_data format

* Update empire/server/api/v2/agent/agent_task_api.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/common/agents.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* working socks after edits

* fixed database holding issue and tests

* fixed deleted contents in invoke-internalmonologue.ps1

* updated poetry.lock with new package

* Don't run the listener for real when in tests

* init

* move client class to a separate package

* remove db file

* fixed ironpython std lib issue if ipy is pre-installed

* module_name optional update

* cleaned formatting

* Update empire/server/core/agent_task_service.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* added multi client socks

* added killing socks thread when agent is killed

* added socks client restart on server reboot

* formatting

* added active jobs to client

* fixed agent crashing when buffer ends

* fixed ironpython job tracking

* kill job thread giving error

* formatting

* fixed killing jobs in ironpython

* fixed pytest

* reverted file removal

* reset test db

* added task functions to python agent

* cleaned up agent functions

* fixed starkiller version

* moved socks client to socks.py

* Update empire/server/common/socks.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* reverted starkiller version

* moved socket import

* added default socks port to description for client

* updated poetry lock and renamed temporary tasks function

* added self tests for jobs

* change jobs class name

* added agent not found tests

* moved db functions to task services

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Minor refactor for agents.py (#482)

* Header keys and values are destructured using a length 2 from the split.
File sizes default to bytes that may get converted to KB and MB if they
exceed 1024.
The logic to calcuate the random sleep duration from the jitter is
extracted into a separate function.

* Updated CHANGELOG.md

* Update stagers with C# and IronPython (#489)

* initial demo for http listener and multi_launcher

* added error response for non-http listeners

* added c# and ironpython stagers and updated stagers to 5.0 format

* fixed errors

* formatting

* removed macroless stager due to being broken

* removed osx_launcher due to redundancy with multi_launcher

* changed python to ironpython on windows_teensy

* updated test

* Update empire/server/listeners/http.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/stagers/windows/backdoorLnkMacro.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/stagers/windows/launcher_lnk.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/stagers/windows/nim.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/stagers/windows/nim.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Update empire/server/stagers/windows/nim.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* removed hardcoded http listener name

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* fixed which varaible gets socks queue saved (#491)

* Fixes for modules requiring files to be uploaded (#490)

* added helper function to handle uploaded files for modules

* updated file encoding for modules

* formatting

* updated to have base64 function called from class download

* Fixed socket staying open after socks server is closed (#492)

* fixed socket staying open after socks server is closed

* added client shutdown function and call in listener

* formatting

* Fixes spurious errors raised on failing to connect to database (#500)

* Created try_connect function to database connection before issuing statements

* makes use of connection instead of engine in tests

* use text for internal_ip so large inputs don't error (#501)

* use text for internal_ip so large inputs don't error

* commit the fix

* Update base.py

* use engine.connect to verify the connection, use the engine itself everywhere else (#503)

* updated powershell agent to properly handle multiple tasking types (#504)

* Fixed issue with C# compilation time at server startup (#510)

* Fixed issue where module and files were throwing errors (#509)

* a few fixes after 4.x merge

* Update the example module templates (#514)

* Update the example module templates

* fix reference to python wiki

* More SOCKS fixes (#515)

* fixed port reuse issue with stale agents

* fixed error handling for sleep in ironpython

* fixed issue where ironpython did not support sleep

* updated lib.zip with updated secretsocks package

* fixed restarting existing socks server

* added socksclient to server restart

* move wrapfunction so its optional, update secretsocks lib.zip, change python to ironpython in c# stager

* reverted renaming languages in c# stager

* formatting

* Minor Client Updates (#521)

* fixed error message displayed for sleep

* removed unused code in usemodule menu

* fixed file upload shortcut and added assembly command

* fixed error when position is less than 2 for files

* add mysql checks

* use sqlite for the install tests

* fix install.sh

* add mysql install for parrot

* add mysql install for parrot

* accidentally committed commented file.

* Bump starkiller to beta4. Fix psransom

* Fixed stageless payloads for python (#520)

* fixed stageless payloads for python

* Update empire/server/common/stagers.py

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* fixed extra space

Co-authored-by: Vincent Rose <vrose04@gmail.com>

* Fix host uniqueness mysql (#525)

* remove some todos

* add blog link

Co-authored-by: Anthony Rose <20302208+Cx01N@users.noreply.github.com>
Co-authored-by: Himadri Bhattacharjee <107522312+lavafroth@users.noreply.github.com>